Papers with reinforcement learning scheme
Atoxia: Red-teaming Large Language Models with Target Toxic Answers (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are still vulnerable to generation safety vulnerabilities. |
| Approach: | They propose a method that A**tacks LLMs with target "toxi" given a particular harmful answer, the method generates a user query and a misleading answer opening to examine the internal defects of a given LLM. |
| Outcome: | The proposed method detects safety risks in open-source models and state-of-the-art models such as GPT-4o. |